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Through evolution, animals have acquired central nervous systems (CNSs), which are ex- 
tremely efficient information processing devices that improve an animal's adaptability to 
various environments. It has been proposed that the process of information maximization 
(infomax 1 ), which maximizes the information transmission from the input to the output of a 
feedforward network, may provide an explanation of the stimulus selectivity of neurons in 
CNSs 2-7 . However, CNSs contain not only feedforward but also recurrent synaptic connec- 
tions, and little is known about information retention over time in such recurrent networks. 
Here, we propose a learning algorithm based on infomax in a recurrent network, which we 
call "recurrent infomax" (RI). RI maximizes information retention and thereby minimizes in- 
formation loss in a network. We find that feeding in external inputs consisting of information 
obtained from photographs of natural scenes into an Rl-based model of a recurrent network 
results in the appearance of Gabor-like selectivity quite similar to that existing in simple cells 
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of the primary visual cortex (VI). More importantly, we find that without external input, 
this network exhibits cell assembly-like and synfire chain-like spontaneous activity 8-10 and 
a critical neuronal avalanche 11-13 . RI provides a simple framework to explain a wide range 
of phenomena observed in in vivo and in vitro neuronal networks, and it should provide a 
novel understanding of experimental results for multineuronal activity and plasticity from 
an information-theoretic point of view. 

Recent advances in multineuronal recording have allowed us to observe phenomena in the 
recurrent networks of CNSs that are much more complex than previously thought to exist. The 
existence of interesting type of neuronal activity, such as patterned firing, synchronization, os- 
cillation, and global state transitions has been revealed by multielectrode recording and calcium 
imaging 1418 . However, in contrast to the rapidly accumulating experimental data, theoretical 
works attempting to account for this wide range of data are developing more slowly. To under- 
stand the behaviour exhibited by recurrent neuronal networks of CNSs, we investigated a network 
employing an RI algorithm that maximizes information retention. The role of RI is to allow a 
recurrent network to optimize the synaptic connection weight in order to maximize information 
retention and thereby minimize information loss by maximizing the mutual information of the 
temporally successive states of the network. 

Here we briefly describe our recurrent network model, leaving the details to the Supplemen- 
tary Notes. In this model, neurons were connected according to the weight matrix W^, and their 
firing states [Xi(t) = 1 (fire) and (not fire)] at time step t are synchronously updated to time step 
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t + 1. The firing state Xi(t+ 1) of neuron i at time step t + 1 is determined stochastically with the 
firing probability 

ft(f+l) = 7 — V' C 1 ) 

1 + exp (- E j Wo-fo(t) - Pi) + **(*)) 

where h,i(t) is the threshold of neuron i, which is adjusted to fix the mean firing probability of 

neuron j to pj, and p max is the maximal firing probability. When p max = 0.5, a neuron fires, on 

average, only once, even if the neuron receives a sufficiently strong excitatory input twice. A small 

value of p max thus makes the firing of the neurons quite unreliable. Thus, p max determines the 

reliability with which a model neuron fires in response to an input. 

To maximize information retention, our recurrent network starts from a random weight 
^initial and develops toward an optimized network with w°f timized (Fig. [Ik), in a manner de- 
termined by the gradient ascent algorithm, 

where I is the mutual information of two successive states of the network and 77 is the learning 
rate. We performed a block of simulation consisting of 10,000-50,000 time steps, updated at 
the end of the block, and then started the calculation for the next block (Fig.[jJ)). 

We first observed the behaviour of this model network under external input. Image patches 
from a photograph preprocessed by a high-pass filter were used as the external input (Fig. [I];). The 
neurons in this network were divided into three groups. 144 on-input and 144 off-input neurons, 
and the 144 output neurons were randomly selected from the network (Fig.[l]ll). Dots with positive 
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and negative values in a randomly selected 12x12 image patch excited the corresponding on-input 
and off-input neurons, respectively. The states of the input neurons were stochastically set to 1 or 
with firing probabilities proportional to the intensities of the corresponding dots, whereas the 
states of the output neurons were not set by the external input. Instead, the firings of these neurons 
were determined by Eq. [T] with p max = 0.95. Initially, the connection weight Wij was a random 
matrix (Fig. [jjl), and therefore the output neurons did not exhibit clear selectivity with respect 
to the external input from the input neurons (Fig.[jjl). After learning, however, the network self- 
organized a feedforward structure from the on-input and off-input neurons to the output neurons 
(Fig.[l£2,d2). Averaging the image patches that evoked firings in an output neuron, we found that 
the output neuron became highly selective to Gabor function-like stimuli (Fig. [jj2), exhibiting 
behaviour quite similar to the selectivity of simple cells in the VI cortex 19 . Our optimization 
algorithm based on RI hence caused the model network to become organized into a feedforward 
network containing simple cell-like output neurons. 

In the simulation described above, the external input was fed into the network with high 
response reliability (p max = 0.95). Next, we examined the evolution of the spontaneous activity in 
the neuronal network without external input. To identify repeated activity in the model network, 
we defined a repeated pattern as a spatial pattern of neuronal firings that occurs at least twice in a 
test block (Fig. |2^). Colouring repeated patterns consisting of > 3 firing neurons in raster plots of 
the network (Fig.|2]:l,c2), we found that the number of repeated patterns increased after learning. 
Several patterns were repeated in a sample of 250 steps as seen in Fig. [2]:2, where the repeated 
patterns are indicated by consistently coloured circles and connected by lines. Moreover, some 
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patterns appeared to constitute repeated sequences. For example, sequence A, composed of the 
magenta, orange, and purple patterns, appears three times in Fig. [2]:2. To be more quantitative, 
we tabulated the numbers of occurrences of repeated patterns and sequences, and compared these 
numbers before and after learning (Fig. [2})). We found that both repeated patterns and repeated 
sequences increased significantly after learning. This indicates that the present algorithm embeds 
not only repeated patterns but also repeated sequences of firings into the network structure as a 
result of the optimization. Thus, when a pattern in a sequence is activated at one step, it is highly 
probable that the next pattern in that sequence will be activated at the next step. This predictability 
means that the state of the network at one time step shares much information with the state at 
the next time step. Hence, we concluded that the repeated activation of an embedded sequence 
is an efficient way to maximize information retention in a recurrent network. These repeated 
patterns and sequences have been experimentally observed in vivo 18 ' 20,21 and in vitro 15 ' 16 , and their 
existence is suggested by the theory of cell assemblies proposed by Hebb 8 and the theory of synfire 
chains proposed by Abeles 9 . We thus see that RI accounts for the appearance of cell assemblies, 
sequences, and synfire chains in neuronal networks. 

We next examined the behaviour of the same spontaneous model in the case that the maxi- 
mal firing probability was small (p max = 0.5). For small p max , the number of identically repeated 
sequences is small, and the network seems to lose structured activity. However, we found char- 
acteristic network activity consisting of firing bursts (Fig. [3^2), which are defined as consecutive 
firing steps that are immediately preceded and followed by "silent" steps, with no firing. We 
found that after learning, the distribution P(s) of the burst size, s, the total number of firings in 
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a burst, obeys a power-law distribution P(s) oc s 7 with 7 « —1.5, whereas, before learning, we 
have P(s) oc exp(— as) (Fig. |3p). This result is consistent with experimental results. Beggs and 
Plenz 11 recorded the spontaneous activity of an organotypic culture from a cortex using multielec- 
trode arrays. Defining an avalanche similarly to our burst, they found that the size distribution of 
avalanches is accurately fit by a power-law distribution with exponent — 1.5 11 . To explain this, they 
argued that a neuronal network is tuned to minimize the information loss and that this is realized 
when one firing induces an average of one firing at the next step. They showed that this condition 
yields the universal exponent —3/2, using the self-organized criticality of the sandpile model 22 ' 23 . 
This condition also holds for the present network, because each neuron with p max = 0.5 after learn- 
ing had two strong input connections and two strong output connections on average (Fig.|3j)2). The 
universal exponent —3/2 was observed in the network for small p max , but for p max = 0.95, the size 
distribution of bursts P(s) in the system did not exhibit a power-law distribution, and displayed 
several peaks, reflecting the existence of stereotyped sequences (see Supplementary Notes). We 
thus conclude that RI embeds information-efficient structures in which one firing induces on aver- 
age one firing at the next step in a network with small p max . 

To reveal the essential mechanism responsible for the behaviour described above, we re- 
turned to the recurrent network with an external input (Fig. In the learning blocks, we repeat- 
edly stimulated neurons 1, 3, and 2 in sequence (Fig. |4^1,bl). In the successive test block, in 
which only neuron 1 was stimulated externally (Fig. [4^2), the firing of neuron 1 was followed by 
spontaneous firings of neurons 3 and 2 (Fig.[4j>2, arrows), because, as we saw above, embedding a 
sequence of firings into the network structure is an efficient way to retain information. In addition, 
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the spontaneous firing of neuron 1 triggers the sequence containing the firings of neurons 3 and 
2 (Fig.[4j)2, double arrows). The form of the weight matrix after learning reveals that a feedfor- 
ward structure starting from neuron 1 was embedded in the network (Fig. |4j:). It is thus seen that 
RI embeds externally input temporal firing patterns into the network by producing feedforward 
structures, and, as a result, the network can spontaneously reproduce the patterns. 

In this study we have found that RI acts to optimize the network structure by maximizing 
the information retained in the recurrent network. Simple cell-like activity, repeated sequences, 
and neuronal avalanches were realized in the model network. These characteristic types of activity 
resulted from the network structure embedded by the optimization algorithm. On the basis of these 
results, we conjecture that RI underlies the neuronal plasticity rule generating these structures 
and activity. We believe that RI will help us to understand the meaning of in vivo and in vitro 
experimental results, particularly to characterize the spontaneous activity of neurons in the context 
of information theory. Our next goal is to derive a plasticity rule in a bottom-up way employing 
RI, and to compare this rule with experimentally obtained plasticity rules. 
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Figure 1 Formation of the feedforward structure through an algorithm based on Rl in 
the model network with external input, (a) Learning changes the initial weight matrix 
^initial t0 ^optimized SQ as tQ max j m j ze ft) e information retained in a recurrent network, (b) 

In the simulation, was updated using Eq.|2]at the end of each block, which consists 
of 10,000-50,000 steps, (c) The original photograph (1024 x 1024) of a pine tree was 
converted to a gray-scaled, high-pass filtered image. Image patches (12 x 12) randomly 
selected from the high-pass filtered image were used as the external inputs to the network 
at each time step. (d1,e1) Initially, 432 neurons were connected according to a random 
weight matrix. Of these neurons 144 were on-input, 144 were off-input, and 144 were 
output neurons. Each of the 144 dots in an image patch was linked to a pair of an on- 
and an off-input neuron in such a manner that the on-input and off-input neurons were 
set to 1 (fire) only when the corresponding dot had a positive and negative sign, respec- 
tively. Output neurons fired spontaneously according to Eq. [T] (d2,e2) After learning, 
feedforward structure from input to output neurons appeared in the model network. (f1 ,2) 
Averaging the image patches that evoked firings of the output neurons revealed that the 
output neurons, which did not exhibit clear selectivity before learning, responded to the 
Gabor-like stimulus after learning. 



Figure 2 Repeated spatial patterns and spatiotemporal sequences occurred frequently 
in the network with p max = 0.95 after learning, (a) We define a repeated pattern as a 
spatial firing pattern that is identically repeated at different time steps. The size of a 
pattern is defined as the number of neurons firing in the pattern. A sequence that contains 
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a particular set of patterns appearing repeatedly in the same temporal order is called a 
"repeated sequence." The size of a repeated sequence is defined as the sum of the 
sizes of the patterns contained in it. (b) The numbers of occurrences of the patterns 
and sequences repeated in the test block (50,000 steps) were compared before and after 
learning. In this histogram, only the sequences with sizes larger than 5/, where I is the 
length of the sequence, were counted, (d ,2) When the repeated patterns in the 50,000 
steps were coloured, it was found that no pattern occurred more than once in this short 
raster plot before learning (c1). By contrast, several patterns appeared multiple times in 
the raster plot after learning (c2). In addition, repeated sequences were found only in the 
raster plot after learning (red stars and blue diamonds). 



Figure 3 Spontaneous activity of the recurrent network with p max = 0.5. (a1 ,2) Individual 
bursts in the spontaneous activity before (a1) and after learning (a2) are indicated by 
different colours. The bursts before learning were short and frequently interrupted by 
steps without firing, whereas the bursts after learning had much longer durations. (b1,2) 
The initial with random weights evolved into a matrix with relatively few strong weights. 
Most rows and columns contained two strong excitatory connections (black dots); that is, 
most neurons had two strong inputs and two strong outputs, (c) Frequency distribution 
P(s) of the burst size plotted as a function of the size, s. The black line corresponds to a 
slope of -1.5. 
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Figure 4 A feedforward structure was embedded in the model network by the temporally- 
structured stimulation. (a1,2) In the learning blocks, the state of neuron 1 was set to 1 
(fire) at random intervals ranging from 50 to 99 steps. The first time step, t, is indicated 
by the arrow in a1 . At t + 2, the state of neuron 3 was set to 1 , and at t + 6, the state 
of neuron 2 was set to 1 . In the test block after learning, only neuron 1 was set to 1 at 
random intervals ranging from 50 to 99 steps (a2). External stimulations are indicated by 
red circles. (b1 , 2) The network activity in an early learning block (b1) and the test block 
(b2). The steps at which neuron 1 was set to 1 are indicated by arrows, and externally 
evoked firings of neurons 1 , 2, and 3 are indicated by red circles. Although the states 
of neurons 2 and 3 were not set from the outside during the test block, neurons 2 and 
3 fired spontaneously six and two steps, respectively, after neuron 1 fired (as indicated 
by orange circles). The sequence of firings embedded by learning was replayed after the 
spontaneous firing of neuron 1 (double arrows), (c) The weight matrix of the network after 
learning (top) and its schematic representation (bottom) indicate a feedforward structure 
which underlies the firing sequence starting from neuron 1 and containing neurons 3 and 
2. 
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